The present invention relates to a method for operating an opto-electronic spatial-resolution sensor device with a sensor surface comprising a large number of photosensitive sensor elements and with a photographic lens system which reproduces a spatial light distribution in surroundings it captures on the active sensor surface, wherein a subset of sensor elements activated in each case for the duration of an assigned activation interval for the integration of incident light to produce a corresponding field of the surroundings, and with a lighting device which is synchronized with the activation intervals to illuminate the surroundings captured by the photographic lens system at least in time and area with light of a wavelength range that can be integrated by the sensor elements.
This kind of method, and these kinds of opto-electronic sensor devices, are generally known from the area of digital imaging. The special control of the sensor elements is referred to as a “rolling shutter”, wherein this expression can refer both to the control method and to a correspondingly operated sensor device. The theoretical operation of the rolling shutter will be first described below.
Digital cameras for recording a rapid sequence of images often work with a matrix-like sensor with a sensor surface which consists of a large number of sensor elements arranged in rows and columns. For the sake of simplicity, in this description of the prior art, reference is made only to these kinds of sensor devices although other arrangements of sensor elements are possible.
A photographic lens system reproduces a spatial light distribution on the sensor surface. In the individual sensor elements, usually special semiconductor arrangements, the striking photons cause isolated electrons to be produced, which are stored in an assigned reservoir over the exposure time of the sensor element. This is called the integration of the striking light in the sensor elements. Following the exposure time, the sensor elements are read out, i.e., the number of stored electrons is determined in spatial reference to the corresponding sensor elements as a measure of the incident photons during the exposure time. Using a data-processing unit which has suitable means of processing such as, for instance, means of digitizing, storing, calculation, etc., a digital representation of the data recorded can be produced which is generally called an “image”. The different points in the image corresponding to the positions of the sensor elements in the sensor surface are called pixels. The digital representation of all the sensor elements used is called the frame.
The sensor elements are reset each time to provide a defined exposure time, which corresponds to emptying the electron reservoir. The exposure time is ended by the re-emptying of the reservoir during read-out. In the context of this application this entire interval is called the activation interval.
The rolling shutter is a special sequence of reset and read-out processes which makes possible a particularly rapid sequence of images. The reset, and therefore, the start of the exposure time for the sensor elements involved, occur each time by row and column in a predetermined cycle, so that all rows and columns are reset successively and in accordance with their spatial arrangement. Therefore, the exposure time for each row and column begins at a different point in time. With a constant offset of one or more rows or columns, the sensor elements are also read out by rows or columns. The exposure time is determined by the amount of the offset between reset and read-out during a predetermined reset and read-out time, i.e., during a predetermined dwell time of the process in a row or column.
The maximum exposure time corresponds to one frame time, i.e., the time required to read out one frame; the minimum exposure time corresponds to the time required to read out one row or column. As long as the exposure time is within these limits, the image sequencing rate is independent of exposure time.
When there is an offset of several columns or rows, there is an overlap between consecutive frames, with the last rows or columns of the preceding frame not having been read out while the first rows or columns of the following frame have already been reset. Accordingly, images can be produced in a continuous manner, which leads to very rapid image sequencing.
On the other hand, this overlap conceals the problem of the inadequate separation of individual frames. In most multimedia applications, this is unnoticeable. However, in measurement arrangements in which different frames have to be compared or processed with one another in some other way, this can lead to difficulties.
A method for monitoring the inside of a motor vehicle is known, for example, from German Patent Document Number DE 100 62 977 A1, in which two images of the inside of a motor vehicle are taken from the same sensor position under different lighting conditions and then subtracted from each other in terms of pixels. The different lighting conditions are produced by appropriately controlling a lighting source for illuminating the monitored inside area so that the lighting source is turned on when one image is captured and turned off when the other image is captured. The subtraction serves to eliminate the background light, i.e. the so-called offset reduction.
The aforementioned patent document does not provide information regarding the special control of the sensor device. However, it would have been particularly desirable to implement such applications with rolling shutter sensor devices because these are the most widely commercially available, and hence, the most cost-effective systems. However, if one wanted to perform this kind of measurement using the conventional rolling shutter method, one could only either reduce the row offset to one row and trigger the lighting source with the frame time, or keep a multi-row offset and discard one overlapping frame between one illuminated and one unilluminated frame. However, the frame time of the computed images, i.e., the time offset between the images with pixel values corresponding to the difference in the corresponding pixels of the original images taken, would be doubled or tripled. This cannot be tolerated for many measuring applications in which rapid processes have to be tracked such as, for example, in the optical control of passive safety systems like airbags.
German Patent Document DE 100 18 366 A1 discloses a method in which a rolling shutter camera is used to monitor the inside of a motor vehicle. In this method a lighting source is triggered with the shutter movement, i.e., the movement of the reset and read-out row over the sensor surface, so that it produces light spots in the monitored space at exactly the time at which those sensor elements are active on which these light spots would have been reproduced in a previously defined nominal condition, e.g., an empty vehicle. An image of the nominal condition is stored for comparison with images currently being taken. If due to a change in the nominal condition, e.g., the entry of a person into the space, the light spots produced are not reproduced on the predetermined sensor elements, this can be ascertained by comparison with the reference image.
The disadvantage to this method is that it is limited to comparing the current situation with a static predetermined condition. There is no evaluation of dynamic processes. In addition, the method actually uses only an extremely small part of the available sensor elements to obtain information, i.e., only those sensor elements on which the light spots generated are reproduced in the nominal condition. The activation of the other sensor elements that is necessary for technical reasons only brings disadvantages in terms of time.
It is an object of the present invention to provide a generic method in such a way that rapid dynamic processes can also be captured using standard commercial rolling shutter sensor devices.
This object is achieved by a spatial-resolution opto-electronic sensor device with a sensor surface comprising a large number of photosensitive sensor elements, and with a photographic lens system which reproduces spatial light diffusion in a surroundings it captures on the sensor surface, wherein a subset of sensor elements is activated in each case for the duration of an assigned activation interval for the integration of incident light to produce a corresponding field of the surroundings, and with a lighting device which is synchronized with the activation intervals to illuminate the surroundings captured by the photographic lens system at least in time and area with light of a wavelength range that can be integrated by the sensor elements, wherein producing a computed image of the surroundings is achieved by activating the sensor surface as a sequence of n-tuples of pairs from activation intervals and assigned subsets to produce a computed image of the surroundings in such a way that n subsets of sensor elements are activated each time within an n-tuple during n activation intervals under different lighting conditions of the surroundings caused by appropriate control of the lighting source, and fields recorded within an n-tuple are processed with one another in a data-processing device.
With this approach it becomes possible to produce n images within one frame time, which represent essentially the same scene under different lighting conditions, and which can be processed with one another in terms of pixels for further evaluation of the situation. The pixels to be processed with one another each time are recorded in immediate chronological proximity. This minimizes errors arising from the recorded scene changing between the exposures of the pixels corresponding to one another in the fields. Accordingly, with the present invention it also becomes possible to examine dynamic processes in which the scene changes within orders of magnitude of time that are not large vis-a-vis the frame time, as must be the case with conventional methods.
In accordance with the present invention, the spatial resolution of the n images recorded during one frame time is reduced compared with a full frame, i.e., an image with one pixel for each sensor element of the sensor surface. Each sensor element contributes to only one of the n images. Apart from that, the n images recorded of essentially the same scene occurring within one frame time are indeed recorded according to the present invention at a minimum interval of time. However, because n spatially different subsets of sensor elements are activated in each n-tuple, slightly different points of the scene are reproduced in the mutually corresponding pixels of the n fields of an n-tuple.
The reduction of spatial resolution and that each sensor element contributes to only one of the n images is not problematic because in practice the resolution of standard commercial sensor elements far exceeds the resolution required for measurement. Specifically, an artificial reduction in resolution is often necessary for rapid image processing. Because the lens system can be adapted to the requirements for measurement, and can therefore also have a lower resolution than the sensor surface, adjacent sensor elements are essentially exposed to the same light incidence and can be practically regarded as pixels of the same scene point.
In accordance with one aspect of the present invention, spatially adjacent subsets of sensor elements are activated within one n-tuple. If the n subsets of sensor elements activated within one n-tuple are in spatial proximity, the spatial offset is minimized so that the differences in the pixels to be processed with one another derive almost exclusively from the different lighting during the particular activation interval.
Within an n-tuple with n>2, this allows the spatially adjacent subsets of sensor elements to be activated in a time sequence which does not correspond to their spatial arrangement. But it is particularly beneficial to activate the spatially adjacent subsets of sensor elements during chronologically sequential activation intervals. Accordingly, it is ensured that the mutually corresponding pixels of the n fields are at a minimal distance from one another in space and time.
In accordance with another aspect of the present invention, the sensor surface comprises a sensor element matrix arranged in rows and columns. This type of sensor is the most common form of sensor, and is, therefore cost-effective. The later processing of the data obtained with this kind of sensor is also particularly simple because the spatial position of each sensor element simply results from the combination of the corresponding row and column.
The subset of sensor elements activated during an activation interval can correspond to a small number of rows or columns in the sensor element matrix, or can correspond to exactly one row or exactly one column.
This provides particularly close agreement in time and space of the n fields of an n-tuple.
It is especially advantageous if all activation intervals have basically the same duration. This makes it easier to control the sensor device. The activation intervals corresponding to one another in the n-tuples in terms of their lighting conditions should at least be of the same length. It may be useful, however, to design activation intervals with different lengths in an n-tuple when there are different lighting conditions.
There are basically three ways to have the fields of one n-tuple processed with one another in time. First, they can be processed in such a way that n images of the surroundings are first produced, in each case, from those fields of all n-tuples of a frame which were recorded in equivalent lighting conditions and these are processed with one another. This requires intermediate storage of the n images. This method is especially useful if there are to be several computer steps which have to capture at least one of the n images in each case. Second, it is also possible to have processing take place immediately after activation. In this method only the data read out from one or a few subsets of sensor elements have to be temporarily stored in the interim. Only the processing result is then constructed as an image, i.e., the computed image. This method is useful, for example, with simple computations such as pixel subtraction in the n=2 case. Finally, it is also possible to produce a combination image of the surroundings first from all fields of all n-tuples of a frame and to have its pixels processed with one another according to their assignment to the n-tuples. This variation requires the smallest change in the control of a conventional rolling shutter sensor device.
To achieve a minimum offset in time and space of the pixels or sensor elements to be processed with one another, and for reasons of higher computing speed, less storage space and reduced loss of resolution, applications in which n=2 are preferred over higher n-tuples.
In many applications such as, for instance, monitoring the inside of a motor vehicle, it is desired that the user be as unaware as possible of the device being used. It is therefore desirable to use a lighting source that emits radiation invisible to the human eye. One particularly preferred form of the present invention is therefore distinguished in that the lighting source emits light in the infrared range.
The method of the present invention can be converted preferably by suitable programming of the control of standard commercial sensor devices, with hardware adaptations being able to be performed as needed and/or other components being able to be provided.
Further details on the invention are derived from the following detailed description and appended diagrams in which preferred embodiments of the invention are illustrated.